Annual Evaluation Report for 
Rural Physics Teacher Resource Agents 



Submitted to: 



Submitted by: 



Covering Period from June 2005 to April 2006 



Sherri L. Fulp 
Eric R. Banilower 



April 2006 



George Amann 

193 Primrose Hill Road 

Rhinebeek,NY 12572 



Jan Mader 

2223 5* Avenue, SW 
Great Falls, MT 59404 



Karen Jo Matsler 
3743 Hollow Creek 
Arlington, XX 76001 

Jim Nelson 

4345 NW 36“^ Drive 

Gainesville, FL 32605 



Warren Hein 

Ameriean Assoeiation of Physies Teachers 

One Physics Ellipse 

College Park, MD 20740-3845 



Horizon Research, Inc. 

326 Cloister Court 
Chapel Hill, NC 27514-2296 




Table of Contents 



Page 

Executive Summary iii 

Introduction 1 

Overview of the AAPT/PTRA Rural Project and Evaluation 1 

Preparation of the PTRAs: The 2005 PTRA Institute 3 

The PTRAs 3 

The Quality and Impacts of the PTRA Institute 4 

2005 Rural Institutes 13 

Participation in the Rural Institutes 13 

The Outreach Participants 17 

Impacts of the Rural Institutes 19 

Summary and Recommendations 28 

Appendix A: Analysis and Reporting of Questionnaire Data 
Appendix B; Questionnaires 



ii 




Executive Summary 



In July of 2005, the AAPT/PTRA Rural Project gathered 73 PTRAs at the University of Utah in 
Salt Lake City for the annual AAPT/PTRA Leadership Institute. The major goal for the summer 
institute is to provide the PTRAs with the knowledge and skills needed to effectively lead 
outreach institutes for rural teachers. The AAPT/PTRA project offered eight workshops during 
the Summer Leadership Institute, covering topics such as inquiry, graphical analysis, physlets, 
and geometric optics. Data collected on a post- institute questionnaire indicate that a majority of 
PTRAs rated the instruction in each workshop as high in quality. The data also show that the 
institute had a positive impact on the PTRAs feelings of preparedness to integrate the content of 
the workshops into the professional development they lead for outreach teachers. Still, even 
after the AAPT/PTRA Leadership Institute roughly 2 out of 5 PTRAs indicated a substantial 
need for additional training in how people learn, technologies for physics instruction, common 
misconceptions/student thinking in physics, and research on and strategies to implement the 
principles of effective professional development. 

The project operated 34 Rural Regional Sites during its fourth year, serving a total of 680 
outreach teachers in rural summer institutes. Fifty-five percent of these teachers attended a 
follow-up session during the school year. In terms of meeting the project’s goal of providing 
three years of professional development for outreach teachers, 144 of the 418 outreach 
participants in sites that have existed for at least three years have attended rural institutes for at 
least three years. However, the data show that the project has made major improvements in 
retaining outreach participants; 50 percent of participants in sites inaugurated in 2003 have 
attended for three years, compared to 15 percent in sites started in 2001 and 24 percent in sites 
started in 2002. 

By linking data across years from an outreach participant questionnaire, HRI examined changes 
in participants’ feelings of preparedness and reports of classroom practices. Analyses show that 
participants’ perceptions of their pedagogical preparedness were significantly higher after one 
year of AAPT/PTRA professional development, with another significant increase after a second 
year of AAPT/PTRA professional development. Participant perceptions of their physics content 
preparedness were significantly higher after one year of professional development, but did not 
increase significantly after a second year of professional development. 

Results from a teacher content knowledge impact study provide additional evidence that the 
AAPT/PTRA rural project is having a positive impact on participants’ physics content 
knowledge. An assessment focusing on ideas in force and motion was administered to all 
teachers attending a summer rural institute, once at the beginning of the institute and again at the 
end of the institute. The study found that teachers scored about 6.5 percentage points higher on 
the post-test than on the pre-test (an effect size of 0.60 standard deviations), getting 
approximately 2 more out of the 28 items correct on the post-test than on the pre-test. 

The AAPT/PTRA Rural Project also appears to be having an impact on how physics is being 
taught. Outreach participants’ report of their frequency of use of investigative teaching practices 
increased significantly after one year of AAPT/PTRA professional development. The extent to 
which teachers create an investigative classroom culture also increased significantly after one 
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year of AAPT/PTRA professional development. In addition, participants’ use of traditional 
teaching practices was significantly lower after two years of professional development than prior 
to participation. 

Recommendations 

The key to the AAPT/PTRA Rural Project’s success is preparing the PTRAs to provide high- 
quality professional development in outreach workshops. One of the strengths of the project is 
that most, if not all, PTRAs have strong physics backgrounds and a great deal of experience 
teaching physics prior to joining the project. Another strength of the project is the collection of 
instructional resources that has been amassed in the creation of the AAPT/PTRA manuals. 

These manuals are the foundation of the outreach workshops, and the outreach participants 
highly value receiving the activities in them. However, leading professional development 
requires skills above and beyond those required to be successful in the classroom and even the 
best activities won’t improve student learning if they are not implemented properly. It is in these 
areas that HRI offers the following recommendations to the project. 

> The project should consider systematically providing additional pedagogical content 
knowledge information with each of the AAPT/PTRA manual activities intended for 
use in the rural institutes. 

In addition to being familiar with the mechanics of implementing an activity in the 
classroom, teachers should know several things to maximize the effectiveness of an 
instructional activity, including: 

• The specific learning goal(s) addressed by the activity; 

• Where in the instructional sequence the activity should be placed (i.e., what 
instruction students should have prior to and after experiencing the activity); 

• The conceptions/misconceptions students are likely to have prior to experiencing the 
activity, as well as which misconceptions the activity confronts; 

• Likely student responses to the activity (both correct and incorrect); and 

• How to wrap-up the activity to increase the likelihood that student sense-making 
occurs. 

Without this type of information, outreach participants may implement an activity without 
having a clear idea of how the activity adds to the conceptual storyline of a unit, resulting in 
“activity for activity’s sake” (i.e., doing hands-on, but not minds-on). Although some 
PTRAs may integrate these aspects into their workshops, systematically building this 
information into the AAPT/PTRA manuals would increase the likelihood that all PTRAs do 
so. 



In addition, the project should consider providing PTRAs with a template cover sheet for 
activities that would include areas for the above information to be completed during the 
outreach workshop. Providing the PTRAs with such a template would emphasize the need 
for them to address these areas in their outreach workshops, and would provide a mechanism 
for the outreach participants to record this important information. 
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> The project should continue to work on developing the PTRAs’ abilities to facilitate 
high-quality professional development. 

Just like students need opportunities to praetice and receive feedback on the skills they are 
expected to master, the PTRAs need opportunities to practice and receive feedback on their 
professional development facilitation skills. Providing the PTRAs with these opportunities 
may require decreasing the Institute’s focus on familiarizing PTRAs with classroom 
activities and increasing the focus on practicing and receiving feedback on leadership skills 
(e.g., leading discussions, informally assessing participants’ understanding, helping outreach 
participants examine student work/focus on questioning strategies). These changes may 
require the project leadership to increase their work with the AAPT/PTRA Leadership 
Institute workshop leaders, helping them incorporate what is known from physics education 
research and the research on effective professional development. 
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Introduction 



This report summarizes the activities and findings of Horizon Research, Inc. (HRI) in its external 
evaluation of AAPT’s Physics Teacher Resource Agents (PTRA) Rural Project since June 2005. 
During the period from June 2005 to April 2006, HRI has: 

• Administered pre- and post-institute questionnaires to the PTRAs attending the 2005 
PTRA Institute; 

• Administered a questionnaire to all teachers who attended the 2005 Rural Regional 
outreach institutes; 

• Interviewed a random sample of nine rural outreach participants; 

• Assisted the project in developing and administering (in a pre -test/post- test design) a 
force and motion content assessment to outreach participants attending Rural Regional 
institutes focused on kinematics and dynamics; and 

• Assisted the project in developing of and administering a student assessment in a study 
of the impact of the AAPT/PTRA Rural Project on students. 

This report is divided into four main sections. The first provides an overview of the 
AAPT/PTRA Rural project and a description of the key questions guiding the evaluation. The 
second presents data on the 2005 PTRA Institute, including PTRAs’ expectations for the 
institute, their perceptions of the quality of the professional development, and the impact of the 
institute on their preparedness to lead rural institutes. The third section reports data collected on 
the rural outreach institutes held during the summer of 2005. These data include a description of 
the 34 rural institutes, the teachers attending them, and the impacts of the institutes on teachers, 
their teaching, and their students. The final section summarizes the key findings and presents 
HRI’s recommendations for the project. 



Overview of the AAPT/PTRA Rural Project and Evaluation 

As stated in the grant proposal, the primary aim of the AAPT/PTRA Rural Project is to “serve 
isolated and neglected rural teachers by building on the experience, expertise, and resources of 
the existing PTRA program. The project will provide opportunities for these teachers to grow 
professionally in physics content, in the use of technology for instruction, and in established 
teaching strategies. Additionally these teachers will develop into a professional and supportive 
network.” To accomplish these goals, the project has adopted a trainer-of-trainers approach. 

The first tier consists of the PTRAs, typically accomplished physics teachers. At annual week- 
long AAPT/PTRA Leadership Institutes, the PTRAs are trained to present workshops on a wide 
variety of physics and pedagogical topics. Most institute workshops are six -hours in length and 
focus on familiarizing the PTRAs with the classroom activities in the AAPT/PTRA manual. The 
institute also provides opportunities for the PTRAs to network and share ideas related to the 
classroom and to workshop leadership. The major goal for the summer institute is to provide the 
PTRAs with the knowledge, experience, and skills needed to effectively lead outreach institutes 
for rural teachers. 
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PTRA-led rural institutes, the second tier, are typically five days long and are intended to focus 
on one or two core physics topics (e.g., kinematics and dynamics). In addition, the project has 
included two, day-long follow-up workshops in the model. These follow-up workshops are 
intended to give the second tier outreach participants an opportunity to revisit concepts and skills 
from the preceding summer institute and to share and reflect on their efforts at incorporating 
what they learned into their classrooms. 

The rural institutes include a strong technology component, seeking to introduce outreach 
participants to a number of the tools that can be used to support physics instruction, including 
graphing calculators and calculator/computer-based laboratory activities. These institutes also 
give rural teachers, who are often the only science teacher in their school, an opportunity to 
network with other science teachers. At this second tier, the project expects to have an impact on 
rural teachers’ understanding of important physics content and their use of effective teaching 
strategies. Further, the project hypothesizes that these changes will lead to impacts in student 
learning. 

The evaluation plan for the AAPT/PTRA Rural project contains both formative and summative 
components, focusing on seven key questions: 

1 . How successful is the project at recruiting and maintaining a cadre of PTRAs, 
including teachers from the areas being served by the rural sites? 

2. To what extent do the AAPT/PTRA Leadership Institutes prepare PTRAs with the 
physics and content-specific pedagogical knowledge needed to present outreach 
workshops? 

3. To what extent do the AAPT/PTRA Leadership Institutes prepare PTRAs with the 
leadership skills and professional development strategies that will enable them to 
design and implement extended high-quality professional development workshops 
that provide in-depth examination of physics content and standards-based teaching 
strategies? 

4. How successful is the project at initiating and maintaining the network of Rural 
Regional Sites, including recruiting, training, and providing on-going support to each 
Rural Regional Coordinator? 

5. How successful is the project in reaching the goal of providing 108 hours of 
professional development (over three years) to under-served rural teachers and what 
is the quality of that professional development? 

6. What impacts does the project have on outreach participants’ attitudes, physics and 
pedagogical preparedness, and classroom practices? 

7. What impact does teachers’ participation in the rural institutes have on their students’ 
achievement in physics? 
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Although the project is not yet complete, data collected during the project’s fourth year provide 
insight into the project’s progress in reaching its goals. 



Preparation of the PTRAs: The 2005 PTRA Institute 

As noted above, the goal of the AAPT/PTRA Leadership Institute is to equip the PTRAs with the 
knowledge and skills necessary to provide high-quality, effective professional development for 
rural teachers. The skills and knowledge needed by the PTRAs include: 

• In-depth understanding of physics content; 

• Knowledge of, and experience using, effective physics teaching strategies; 

• Knowledge of effective professional development strategies/adult learning theory; and 

• Skill at designing and implementing high-quality professional development. 

One of the major strengths of the project is that most, if not all, PTRAs join the project with a 
good deal of expertise in the first two areas (the project seeks to attract expert teachers to join the 
PTRA ranks). Thus, the Leadership Institute can place a greater emphasis on the latter two 
areas, which are often difficult for teachers transitioning to the role of professional development 
providers to master. 

The Leadership Institute incorporates a variety of activities, including presentations by physics 
professionals, a session in which PTRAs share a favorite classroom activity or demonstration, 
and opportunities for networking. However, the main component of the institute is a set of 
workshops that focus on various physics topics, technological tools (e.g., graphing calculators), 
and/or teaching strategies (e.g., inquiry, modeling). These workshops are developed by selected 
PTRAs, members of the project leadership, and/or other interested and knowledgeable members 
of the physics education community. Most of these workshops are six-hours long, though a few 
are as few as three or as many as twelve-hours in length. The workshops provide opportunities 
for the PTRAs to experience a sample of the classroom activities included in the AAPT/PTRA 
manuals, and a forum to discuss physics content, classroom practices, and issues of leadership. 

In July of 2005, the project gathered 73 PTRAs at the University of Utah in Salt Lake City for 
the Leadership Institute. This year, four new PTRAs, recruited from the ranks of rural outreach 
participants, attended the institute. The project offered eight workshops during the 2005 PTRA 
Institute, covering topics such as inquiry, graphical analysis, physlets, and geometric optics. 

This section of the report focuses on the quality and impact of the Leadership Institute using data 
collected from the pre- and post-institute questionnaires. 



The PTRAs 

The pre-institute questionnaire gathered a variety of data from the PTRAs, including 
demographic characteristics and information on their learning needs as professional development 
providers. Sixty PTRAs responded to the pre-institute questionnaire, a response rate of 83 
percent. Table 1 shows the demographic characteristics of the responding PTRAs. Sixty-eight 
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percent of the respondents were male, and all were white. Half taught in suburban schools, and 
35 percent taught in rural schools. Seventy-eight percent taught physics and/or physical science 
during the 2004-05 academic year and nearly three-quarters have over 20 years of teaching 
experience. The majority of attendees became PTRAs prior to 1997; 14 percent have become 
PTRAs since the beginning of the rural project in 2002. 



Table 1 

Demographic Data for PTRAs Attending the 2005 Summer Institute 





Percent of Responding PTRAs 
(N = 60) 


Physics/Physical Science in 2004-05 Academic Year 


78 


Gender 




Male 


68 


Female 


32 


Race/Ethnicity 




White 


100 


Asian or Pacific Islander 


0 


American Indian or Alaskan Native 


0 


African-American 


0 


Hispanic 


0 


Other 


0 


Location of School 




Suburban 


50 


Rural 


35 


Urban 


15 


Year Originally Became a PTRA 




1985-1988 


29 


1992-1996 


29 


1997-2001 


29 


2002-2005 


14 


Membership in Professional Organizations 




AAPT 


95 


NSTA 


55 


Years of Physics/Physical Science Teaching Experience 




0-5 Y ears 


3 


6-10 Years 


7 


11-15 Years 


5 


16-20 Years 


12 


21 or More Years 


72 



The Quality and Impacts of the PTRA Institute 

PTRAs ’ Needs and Expectations 

Knowing what participants’ needs and expectations are for a professional development 
experience can provide valuable insight into their perceptions of the quality of that experience. 
The pre-institute questionnaire asked PTRAs the extent to which additional training in a number 
of areas would help them become more effective professional development providers. About 
half of the PTRAs indicated a need for at least a moderate amount of additional training in the 
research on how people learn, principles of effective professional development, and common 
misconceptions/student thinking in physics. (See Table 2.) Relatively few PTRAs indicated a 
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moderate or greater need for additional training in physics content (20 percent) or activities for 
physics instruction (30 percent). 



Table 2 

PTRAs’ Perceived Needs for Additional Leadership Training 





Percent of Responding PTRAs 
(N = 60) 


No 

additional 

training 


A little 
additional 
training 


Moderate 
amount to 
a lot of 
additional 
training 


The research on how people leam 


9 


41 


50 


The research on the principles of effective professional development 


5 


47 


48 


The research on common misconceptions/student thinking in physics 


2 


51 


47 


Strategies for implementing the principles of effective professional 








development in workshops for other teachers 


5 


53 


42 


Technologies for physics instruction 


5 


54 


41 


Activities for physics instruction 


3 


67 


30 


Physics content 


15 


65 


20 



PTRAs’ responses to an open-ended item on the questionnaire regarding what they hoped to gain 
from the Leadership Institute provided further insight into their perceived needs for leadership 
training. The most common response, given by 55 of the 58 PTRAs answering this question, 
was becoming familiar with new activities and/or learning new strategies. Of these 55 
respondents, 29 specifically mentioned activities/strategies to use in leading workshops, 3 
mentioned activities/strategies for use in their classrooms, 15 referred to both workshop and 
classroom use, and 8 were nonspecific. Learning physics content and networking with their 
colleagues were also mentioned by quite a few PTRAs (15 and 14, respectively). Taken as a 
whole, these data indicate that many PTRAs entered the institute focusing on what would benefit 
them both as a workshop leader and as a teacher. As two PTRAs described their goals: 

7 hope to gain added information about physics and physics teaching. I would also like 
to gain information on how to plan and conduct a PTRA workshop. I am also looking 
forward to the networking with physics teachers and exchange of ideas and information. 
(New PTRA) 

I want to improve and enhance my understanding of common misconceptions and learn 
effective methods to overcome these problems. I want to learn and develop new activities 
to bring a better understanding of physics concepts to my students and workshop 
participants. (Veteran PTRA) 

PTRAs ’ Experiences at the AAPT/PTRA Leadership Institute 

The main vehicle for preparing the PTRAs as professional development providers is the institute 
workshops. The project offered seven workshops during the 2005 PTRA Institute, plus a 
leadership session. The post-institute questionnaire asked the PTRAs a number of questions 
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about the institute, including which workshops they participated in and the quality of those 
workshops. Fifty-seven PTRAs returned the post- institute questionnaire, a response rate of 78 
percent. 

Table 3 shows the title, duration, and percentage of PTRAs taking each workshop. The 
workshops taken by the greatest number of PTRAs were Ranking Tasks/TlPERs (“Tasks Inspired 
by Physics Education Research”) and Inquiry, with 86 and 84 percent participation, respectively. 



Table 3 

Participation Data for Workshops Offered during the 2005 PTRA Institute 





Duration 

(Hours) 


Percent of Responding PTRAs 
(N = 57) 


Ranking Tasks/TlPERs 


6 


86 


Inquiry 


6 


84 


Make and Take 


3 


70 


Leadership 


3 


67 


Graphical Analysis 


6 


56 


CASTLE 


12 


52 


Geometric Optics 


6 


52 


Physlets 


6 


52 



The PTRAs clearly found the workshops worthwhile; a majority of PTRAs rated the quality of 
instruction in each workshop as high in quality. (See Table 4.) Over 90 percent of responding 
PTRAs rated Ranking Tasks/TlPERs, CASTLE, and Make and Take highly. 



Table 4 

PTRAs Rating Workshop Instruction as High Quality^ 





N* 


Percent of Responding PTRAs 


Ranking Tasks/TlPERs 


46 


98 


CASTLE 


28 


96 


Make and Take 


37 


92 


Physlets 


27 


89 


Inquiry 


46 


87 


Graphical Analysis 


28 


86 


Leadership 


34 


74 


Geometric Optics 


28 


68 



^ Includes those who rated the workshop a 4 or 5 on a five-point scale from 1 “poor” to 5 “excellent.” 

^ By design, not all PTRAs participated in each workshop; the total number responding for each workshop 



is included in the table. 



The PTRAs were also asked about the extent to which the institute focused on various goals 
related to their preparation as professional development providers. Because the project has been 
working to increase its focus on preparing PTRAs as professional development providers, rather 
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than just as classroom teachers, Table 5 shows responses to this set of questions from both the 
2005 and 2004 institutes. The data indicate that the projeet has been sueeessful in shifting the 
emphasis of the institute; signifieantly more PTRAs in 2005 than in 2004 reported learning about 
the researeh on how people learn, the researeh on eommon miseoneeptions in physios, researoh 
on the principles of effeotive professional development, and strategies for implementing those 
prinoiples into workshops (with effeot sizes' ranging from 0.33 to 0.72). 



Table 5 

PTRAs’ Indicating that each of the 

Following Occurred to a Large Extent^ at the Summer Institute, by Year 





Percent of Responding PTRAs 


Effect 

Size 


2004 
(N = 56) 


2005 
(N = 57) 


Learned about the research on how people learn* 


25 


60 


0.72 


Learned about the research on common misconceptions in physics* 


39 


65 


0.53 


Learned about the research on the principles of effective professional 








development * 


33 


53 


0.41 


Learned strategies for implementing the principles of effective 








professional development into workshops for other teachers* 


54 


70 


0.33 


Gained activities for physics instruction 


86 


89 


— 


Gained experience with technologies for physics instruction 


68 


64 


— 


Learned physics content 


53 


61 


— 



^ Includes those who rated the item 4 or 5 on a five-point scale from 1 “not at all” to 5 “to a great extent.” 

* Percentage of 2005 respondents indicating the occurrence of the topic to a large extent is significantly different than the 



percent of 2004 respondents (two-tailed z-test, p < 0.05), 



As can also be seen in Table 5, nearly all PTRAs indieated that the institute focused heavily on 
activities for physics instruction. However, the PTRAs appear to have gotten the message that 
the aetivities in the institute were meant to help them as professional development providers. On 
an open-ended item on the post-institute questionnaire, the PTRAs were asked to explain what 
they had gained as a result of their participation in the summer institute that would help them in 
their role as professional development providers. Twenty-nine of the 42 PTRAs responding to 
this item indieated they had gained activities and strategies. Of these 29 PTRAs, 18 speeifically 
mentioned gaining activities/strategies for leading workshops, nine mentioned learning 
activities/strategies in general and, 2 mentioned learning aetivities/strategies for teaching 
students. Examples of PTRAs’ responses include: 

I feel I’ve gained many more strategies for helping other teachers not only teach physics 
and physical science more effectively but also to help them learn physics content 
themselves. (Veteran PTRA) 



' Effect sizes of about 0.20 are typically considered small, 0.50 medium, and 0.80 large. Cohen, J. (1988). 
Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Lawrence Erlbaum Associates. 
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1 developed further understanding in the use of inquiry learning in my own workshops. I 
obtained additional materials to assist in preparing other teachers to become more 
effective. (Veteran PTRA) 

Other responses to the open response item included the opportunity to network with other 
PTRAs and increased confidence/enthusiasm, noted by 5 and 4 PTRAs, respectively. 

Impacts of the PTRA Institute 

By comparing responses from the pre- and post-institute questionnaires, HRI is able to examine 
the impact of the institute on the PTRAs’ perceptions of their preparedness to lead outreach 
participant workshops.^ The questionnaires probed two aspects of the PTRAs’ preparedness to 
apply what they learned in the Leadership Institute to their outreach workshops. The first aspect 
was their preparedness to deepen outreach participants’ understanding of the physics content 
related to the workshop. The second was their preparedness to train outreach participants to 
implement the activities/instructional strategies from the workshop in their own classrooms. 

As can be seen in Table 6, for each of the seven workshops the change in preparedness to deepen 
teachers’ understanding of related physics content was greater for PTRAs participating in a 
workshop than non-participants. Particularly large gains were seen for CASTLE, Inquiry, and 
Ranking Tasks/TIPERS. For example, prior to the institute, 29 percent of PTRAs felt well 
prepared to deepen teachers understanding of the physics content related to Ranking 
Tasks/TIPERS. After the institute, 89 percent of the PTRAs attending that workshop, compared 
to 14 percent of the PTRAs not attending that workshop, felt well prepared to deepen teachers’ 
understanding of related content. 

Although PTRAs participating in the Physlets workshop had larger gains in preparedness than 
non-participants, nearly half of the participating PTRAs did not feel well prepared after the 
workshop. This result may be due to the nature of the Physlets workshop, which centers on 
computer simulations related to a wide variety of physics topics. The relatively high 
preparedness of non-participants in CASTLE and Graphical Analysis, both before and after the 
institute, is likely due to many of the PTRAs attending these workshops in previous years and 
maintaining their feelings of preparedness. 



^ HRI was able to match the pre- and post-questionnaire responses of 52 PTRAs; some PTRAs submitted one, but 
not both questionnaires. 
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Table 6 

PTRAs Feeling Well Prepared^ to Deepen Teachers’ 
Understanding of Related Physics Content, by Workshop Participation 





Percent of Respondin 


g PTRAs 


N* 


Pre 


Post 


CASTLE (Electricity)* 








Participants 


26 


23 


96 


Non-Participants 


24 


75 


75 


Geometric Optics * 








Participants 


23 


35 


65 


Non-Participants 


21 


29 


38 


Graphical Analysis* 








Participants 


24 


63 


96 


Non-Participants 


22 


77 


77 


Inquiry* 








Participants 


41 


32 


88 


Non-Participants 


6 


33 


17 


Make and Take* 








Participants 


35 


57 


89 


Non-Participants 


11 


55 


36 


Physlets* 








Participants 


25 


20 


56 


Non-Participants 


19 


37 


42 


Ranking Tasks/TIPERS* 








Participants 


45 


29 


89 


Non-Participants 


7 


29 


14 



^ Includes those who rated the item a 4 or 5 on a five-point scale from 1 “not adequately prepared” to 5 
“very well prepared.” 



* By design, not all PTRAs participated in each workshop; the total number responding for each 
workshop to both the pre- and post-institute questionnaires is included in the table. 

* The change in participants’ perceptions of preparedness is statistically different than non-participants’ 
change (Ordinal Regression, p < 0.05). 



When PTRAs did not feel well prepared to apply what they learned in a workshop to deepen 
teaehers’ understanding of related physies eontent, the post-institute questionnaire asked them to 
explain why the session did not better prepare them. The most eornmon response, given by 12 of 
the 26 PTRAs replying to this question, was that they needed more time to beeome familiar with 
the workshop. Of these 12, 10 speeifieally mentioned needing more experience with the 
activities and strategies presented in the workshop. As two PTRAs wrote; 

This was my first AAPT[/PTRA] meeting and was the first time I saw the AAPT[/PTRA] 
material. We could not cover all the material in the handouts. I will just need time to 
look over the material and do some of the experiments myself. I wouldn ’tfeel 
comfortable unless I had done them myself. (New PTRA) 

Basically my sense of preparedness is because I have not yet applied what I have learned. 
I am sure that once I apply the Salt Lake City workshop program I will be secure in 
success. (Veteran PTRA) 

HRI also compared the PTRAs’ preparedness before and after the institute to train teachers to 
implement the activities/instructional strategies from the workshop in their own classrooms. 



Horizon Research, Inc. 



9 



April 2006 







PTRAs participating in each of the seven workshops had significantly greater inereases in their 
pereeptions of preparedness to help teaehers use the resourees from a workshop in their 
instruetion than non-partieipants. (See Table 7.) As was the ease with deepening teaehers’ 
eontent knowledge, PTRAs in the CASTLE, Inquiry Tasks, and Ranking Tasks/TIPERS 
workshops had particularly large ehanges relative to non-participants. 



Table 7 

PTRAs Feeling Well Prepared^ to Train Teachers to Use the 
Resources in their Classroom Instruction, by Workshop Participation 





Percent of Responding PTRAs 


N* 


Pre 


Post 


CASTLE (Electricity)* 








Participants 


27 


15 


96 


Non-Participants 


23 


70 


74 


Geometric Optics * 








Participants 


25 


40 


80 


Non-Participants 


21 


24 


43 


Graphical Analysis* 








Participants 


24 


58 


88 


Non-Participants 


20 


75 


75 


Inquiry* 








Participants 


41 


24 


85 


Non-Participants 


5 


20 


20 


Make and Take* 








Participants 


32 


53 


78 


Non-Participants 


10 


40 


30 


Physlets* 








Participants 


24 


13 


63 


Non-Participants 


18 


39 


39 


Ranking Tasks/TIPERS* 








Participants 


44 


27 


95 


Non-Participants 


6 


17 


0 



^ Includes those who rated the item a 4 or 5 on a five-point scale from 1 “not adequately prepared” to 5 
“very well prepared.” 



* By design, not all PTRAs participated in each workshop; the total number responding for each 
workshop to both the pre- and post-institute questionnaires is included in the table. 

* The change in participants’ perceptions of preparedness is statistically different than non-participants’ 
change (Ordinal Regression, p < 0,05), 



Again, the most common reason PTRAs did not feel well prepared to help teaehers implement 
resourees in their classroom instruction, given by 16 of the 26 respondents, was that they needed 
more time to beeome familiar with the workshop. Eleven of these 16 speeifieally mentioned 
needing more experienee with the aetivities and strategies presented in the workshop. As two 
PTRAs wrote: 

I need to read the manual to feel comfortable with the ideas. (Veteran PTRA) 

I want to use these in my own classes for a while before I encourage others to do the 
same. (Veteran PTRA) 
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The PTRAs were also asked a series of questions on the pre- and post-institute questionnaires 
regarding their preparedness to lead professional development. These items were eombined into 
a eomposite variable to reduee the unreliability assoeiated with individual survey items. 
(Definitions of the composite variable, a description of how it was created, and reliability 
information are included in Appendix A.) The composite has a minimum possible score of 0 and 
a maximum possible score of 100. A score of 0 would indicate that a PTRA selected the lowest 
response option for each item in the composite, whereas a score of 100 would indicate that a 
PTRA selected the highest response option for each item. 

The composite measures PTRAs’ feelings of preparedness to do a number of activities in their 
workshops, such as helping other teachers (1) develop an understanding of physics concepts, (2) 
examine science teaching strategies, and (3) develop questioning strategies to elicit student 
understanding. By linking PTRAs’ responses from the pre- and the post-institute questionnaires, 
HRI is able to examine changes in this composite score. As can be seen in Table 8, PTRAs’ 
perceptions of their preparedness to provide professional development increased significantly 
after participating in the summer institute. The effect size for the change is very large, 1.01 
standard deviations. 



Table 8 



Composite: PTRAs’ Perceptions of their 
Preparedness to Provide Professional Development 



(N = 52) 


Mean 


Standard Deviation 


Pre 


68.16 


15.43 


Post* 


80.82 


12.96 



* Post-questionnaire composite score significantly greater than pre-questionnaire 
composite score (one-tailed dependent samples t-test, p < 0,05). 



On both the pre- and post-institute questionnaire, PTRAs were asked the extent to which they 
needed leadership training in each of a number of areas. It was hypothesized that participation in 
the AAPT/PTRA Leadership Institute would result in a decreased need for additional training. 

As can be seen in Table 9, after attending the institute, there appears to be a downward trend in 
the percent of PTRAs perceiving a need for additional training in several areas, although the 
differences were not statistically significant. It is also noteworthy that roughly 2 out of 5 PTRAs 
indicated a substantial need for additional training in how people learn, technologies for physics 
instruction, common misconceptions/student thinking in physics, and research on and strategies 
to implement the principles of effective professional development. This finding indicates that 
the project may need to take additional steps to adequately prepare all PTRAs for their role as 
professional development providers. 
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Table 9 



PTRAs Indicating a Substantial Need^ for Additional Leadership Training 





Percent of Responding PTRAs 
(N = 51) 


Pre* 


Post 


Difference* 


The research on how people leam 


48 


38 


-10 


Technologies for physics instruction 


45 


40 


-5 


Activities for physics instruction 


30 


26 


-4 


The research on the principles of effective professional development 


47 


45 


-2 


The research on common misconceptions/student thinking in physics 


44 


42 


-2 


Strategies for implementing the principles of effective professional 








development in workshops for other teachers 


43 


43 


0 


Physics content 


18 


18 


0 



^ Includes those who rated the item 3 or 4 on a four-point scale from 1 “I don’t need any additional training” to 5 “I could use a 
lot of additional training.” 



* The pre-institute percents shown here are different than those shown in Table 2 as this table includes responses from only 
those PTRAs that completed both a pre- and post-institute questionnaire. 

^ Post-institute responses are not significantly different than pre-institute responses (Wilcoxin Signed Ranks Test, p > 0.05). 



Data from PTRAs’ responses to an open-ended item on the post- institute questionnaire that 
asked what additional training would be helpful in preparing them for their role as a professional 
development provider also indieate that PTRAs would like more training in many of these areas. 
The most eommon request was for more aetivities and strategies, given by 27 of the 42 PTRAs 
responding to this question; 12 of these respondents speeifieally mentioned wanting 
aetivities/strategies to use in leading a workshop. Additionally, 8 of the 42 respondents 
requested more information on how students learn physies/eommon miseoneeptions, 5 asked for 
more physies eontent knowledge, 4 asked for information on physios eduoation researoh in 
general, and 4 asked for more time to interaot with their peers at the institute. Examples of 
oomments made by PTRAs inolude: 

I feel inadequately prepared to help other teachers understand how people learn and 

common misconceptions students have in physics. (Veteran PTRA) 

I can always use help with using activities and strategies for professional development. 

(Veteran PTRA) 

As a final indioator of the impaot of the institute, PTRAs were asked in what ways the outreaoh 
workshops they lead will be different as a result of attending the summer institute in Salt Lake 
City. The most eommon response to this open-ended item, given by 26 of the 49 respondents, 
was that they would ohange the pedagogy used in their workshops; half of these speeifieally 
mentioned adding or inoreasing the foous on inquiry in their workshops. The seeond most 
eommon response to this open-ended item, given by 18 respondents, was that the PTRAs would 
add more aetivities to their workshops. It was not elear from the responses if they intended to 
inerease the number of aetivities they use in a workshop or replaee some of the aetivities with 
new ones. Thus, the projeet may want to reemphasize to the PTRAs that new aetivities are 
meant to diversify, not overload, the workshops. Thirteen PTRAs mentioned adding ranking 
tasks/TIPERs in partieular. Examples of PTRAs responses inelude: 
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1 will add inquiry and ranking task ideas in any workshop I give. (Veteran PTRA) 



1 will emphasize more methods of teaching and try to model them. (Veteran PTRA) 

More inquiry activities with discussion of value. Already do this but more emphasis on 
process. (Veteran PTRA) 

Greater emphasis on misconceptions and processes of thinking when developing 
concepts. (Veteran PTRA) 



2005 Rural Institutes 

The main goal of the AAPT/PTRA Rural Projeet is to improve the teaching and learning of 
physics/physical science in rural classrooms via the AAPT/PTRA Rural Regional sites. The 
project’s model is for each site to host a 30-hour summer institute and two, six-hour follow-up 
sessions during the school year. The summer institute is intended to focus on a small number of 
physics topics and provide outreach participants the opportunity for in-depth study of both the 
physics content and teaching strategies. The two follow-up sessions are intended to give 
outreach participants an opportunity to revisit the topic and reflect upon their attempts to 
incorporate what they learned into their classroom teaching. 

Each Rural Regional Site is hosted by a local university and has a designated Rural Regional 
Coordinator, typically a member of the university’s physics department. The coordinator’s 
responsibilities include recruiting outreach participants, arranging facilities and equipment for 
the institutes, and managing all of the necessary paperwork. This arrangement makes it possible 
for the PTRAs to focus their energies on designing and implementing the professional 
development. 

This section of the report examines teacher participation in, and the impacts of, the rural 
institutes. Data come from project records of participant attendance; a questionnaire 
administered to all rural institute participants; a content assessment administered to outreach 
teachers attending institutes focusing on kinematics and dynamics; interviews with a sample of 
outreach participants; and a student impact study. 



Participation in the Rural Institutes 

The AAPT/PTRA Rural Project operated 34 rural regional sites^ during its fourth year. Three of 
the sites were continuations of “prototype” institutes created to test the logistics of this model 
prior to NSF funding, 1 site was initiated in the project’s first year, 7 in the second year, 14 in the 
third year, and 9 in the fourth year of NSF funding. Table 10 shows the number of outreach 



^ The Colorado School of Mines hosted two institutes in 2005 and is counted twice. 
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participants attending each of the rural institute summer and follow-up sessions.'* A total of 680 
teachers attended the Rural Regional Site summer institutes; 55 pereent of these teachers 
attended a follow-up session during the sehool year.^ Fifty-four percent of the outreaeh 
partieipants reaehed the project’s goal of 36 hours of professional development during a year. 



Participation data come from AAPT/PTRA Rural project records and are current as of March 30, 2006. 
^ Additionally, seven teachers attended follow-up sessions without having attended a summer institute. 
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Table 10 



Outreach Participants Attendance, by Rural Regional Site 



Rural Regional Site 


Number of Outreach Participants 


Summer 
Institute 
and Follow- 
Up Session* 


Summer 

Institute 

Only 


Follow- 

Up 

Session 

Only* 


At Ueast 36 
Hours of 
Professional 
Development this 
Year 


Auburn University 


0 


9 


0 


0 


Bismarck State College 


35 


14 


1 


35 


Brigham Young University 


12 


3 


1 


10 


Central Pennsylvania 


10 


6 


0 


10 


Coastal Carolina University 


0 


18 


0 


0 


Colby College 


13 


6 


0 


12 


Colgate University 


16 


10 


1 


16 


Colorado School of Mines (Kinematics and Dynamics) 


6 


16 


0 


6 


Colorado School of Mines (Momentum and Energy) 


5 


9 


1 


5 


Eastern Kentucky University 


19 


9 


0 


17 


Emporia State University 


9 


7 


2 


8 


Frostburg State University 


7 


4 


0 


6 


Georgia College and State University 


18 


6 


0 


18 


Gonzaga University 


0 


13 


0 


0 


Higher Education Consortium of Central California 


17 


8 


1 


17 


Idaho State University 


28 


17 


1 


28 


Illinois State University 


0 


19 


0 


0 


James Madison University 


13 


5 


0 


13 


Lee College 


14 


7 


1 


14 


Mississippi State University 


0 


9 


0 


0 


Montana State University 


11 


7 


0 


11 


Ohio State University 


0 


16 


0 


0 


Saginaw Valley State University 


23 


3 


0 


23 


Santa Fe Community College 


7 


0 


0 


7 


Sewanee: University of the South 


17 


4 


0 


17 


South Dakota State University 


14 


8 


0 


13 


State University of New York - Fredonia 


0 


29 


0 


0 


Texas A&M University 


0 


26 


0 


0 


Texas Tech University 










University of Arkansas 


10 


8 


0 


18 


University of Dallas 


8 


2 


1 


7 


University of Pittsburg - Bradford 


9 


9 


0 


9 


University of Wisconsin - River Falls 


13 


3 


1 


11 


Youngstown State University 


11 


4 


0 


11 


Total Number* 


374 


322 


13 


367 


The duration of follow-up sessions ranged from 2 to 12 hours; the median duration was 6 hours. 



* Twenty-two outreach participants attended professional development at more than one rural regional site; these 
individuals are represented in the counts for each site attended. Thus, the totals slightly overestimate the number of 
individual teachers participating in the project. 
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Interviews with a sample of nine outreaeh partieipants*’ shed some light on the reasons why 
teaehers do and do not attend follow-up workshops. Six of the 9 interviewees had attended a 
follow-up session during the sehool year, a somewhat higher rate than the 54 pereent of summer 
institute attendees that attended a follow-up workshop. Of these 6, 3 stated that they attended the 
follow-ups beeause it was a requirement of the projeet. The other three attended beeause they 
found the summer institute to be highly worthwhile. As one outreaeh partieipant said: 

[I attended] because of the effectiveness and degree to which the main courses were good 

for me. I got a lot out of the summer sessions, and I wanted more. 

Of the 3 partieipants who did not attend a follow-up session, 2 stated that the meetings were too 
far from their homes and ineonvenient to attend; 1 said that a family funetion eonflieted with the 
meeting. 

The results of the interviews are eonsistent with data eolleeted in previous years. Finding a 
meeting time that works for all partieipants at a rural site has been ehallenging. Similarly, the 
faet that some partieipants are loeated hundreds of miles from the rural site makes travel for a 
one-day workshop diffieult. However, as some sites have higher rates of partieipation in the 
follow-up sessions, the projeet may want to ereate a meehanism for site leaders to share 
strategies they have found effeetive for boosting attendanee. 

In addition to the goal of providing at least 36 hours of professional development to partieipants 
per year, the projeet has the larger goal of providing at least 108 hours of professional 
development over the eourse of three years. By eombining partieipant data provided by the 
projeet from the past several years, it is possible to examine the projeet’s progress towards 
reaehing this goal. As ean be seen in Table 11, 144 of the 418 outreaeh partieipants in sites that 
have existed for at least three years have attended rural institutes for at least three years. The 
data also show that the projeet has made major improvements in retaining outreaeh partieipants; 
50 pereent of partieipants in sites inaugurated in 2003 have attended for three years, eompared to 
15 pereent in sites started in 2001 and 24 pereent in sites started in 2002. 



® HRJ attempted to interview 10 randomly seleeted outreaeh partieipants. Only 5 participants from the original 
sample of 10 agreed to be interviewed. HRI contacted another 10 participants, which yielded 4 additional 
interviews. The 9 interviewees represent eight different rural sites. Two of the participants had attended an institute 
for the first time in 2005; 7 had also participated in previous years. 
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Table 11 



Retention Rates for Outreach Participants, by Inaugural Year of the Site 





Inaugural Year of Rural 


Regional Site 






2001 ^ 


2002 


2003 


2004 


2005 


Total 


Number of Participants Attending 

1 Rural Institute 


82 


27 


56 


190 


137 


492 


2 Rural Institutes 


45 


12 


52 


208 


— 


317 


3 Rural Institutes 


21 


12 


109 


— 


— 


142 


4 Rural Institutes 


2 


0 


— 


— 


— 


2 


Total 


150 


51 


217 


398 


137 


953 


Percent Attending for at Least Three Years 


15 


24 


50 


— 


— 


— 



^ “Prototype” sites 



The Outreach Participants 

A questionnaire administered at the beginning of eaeh rural institute eollected a variety of 
information on the outreaeh partieipants. Sinee the questionnaires were administered on-site at 
the beginning of eaeh institute, a 100 pereent response rate was aehieved. As can be seen in 
Table 12, about half of the outreach participants were female and nearly all were white. Eighty- 
three percent taught high school during the 2004-05 academic year. Sixty-three percent taught 
physics, and 60 percent taught physical science (88 percent taught physics and/or physical 
science). Given the project’s target audience of rural teachers, it is not unexpected that 75 
percent of the outreach participants taught other science subjects, and more than 1 in 4 taught 
non-science classes. 

Table 13 shows the number of semesters of college coursework completed by the outreach 
participants. Forty-five percent of the outreach participants have taken eight or more college 
semesters of physics/physical science; nearly one-third have taken three or fewer semesters. 
Outreach participants were also asked to identify those disciplines in which they are certified to 
teach at the secondary level. As can be seen in Table 14, 77 percent are certified to teach 
physics/physical science. It is important to note that several participants did not answer the 
questionnaire items regarding college coursework and certification to teach. If participants 
skipped items for which they did not have coursework/certification, these data would 
overestimate participants’ qualifications. 
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Table 12 



Demographic Data for Outreach Participants 





Percent of Outreach Participants 
(N = 659) 


Gender 




Male 


49 


Female 


51 


Race’^ 




White 


95 


Black or African-American 


3 


American Indian or Alaskan Native 


1 


Asian 


1 


Hispanic or Latino 


1 


Native Hawaiian or Other Pacific Islander 


0 


Grade Level Taught 




High School 


83 


Middle School 


24 


Elementary School 


2 


Not a Classroom Teacher 


4 


Prior Teaching Experience 




0-2 Years 


14 


3-5 Years 


16 


6-10 Years 


22 


1 1-20 Years 


31 


2 1 or More Y ears 


17 


Teaching Assignment Includes^ 




Physics 


63 


Physical Science 


60 


Other Science 


75 


Non-Science 


29 



^ Percents may add to more than 100 as participants could select more than one category. 



Table 13 



Outreach Participants’ College Course work 





N 


Percent of Outreach Participants 


0 

Semesters 


1-3 

Semesters 


4-7 

Semesters 


8 or More 
Semesters 


Life Science/Biology 


602 


9 


26 


14 


51 


Chemistry 


616 


7 


21 


24 


48 


Physics/Physical Science 


635 


6 


25 


25 


45 


Mathematics 


607 


2 


25 


31 


42 


Earth/Space Science 


589 


21 


40 


19 


21 


Engineering/T echnology 


548 


39 


34 


12 


15 



Table 14 

Outreach Participants Certified to Teach Each of the Following Subjects 





N 


Percent of Outreach Participants 


Physics/Physical Science 


610 


77 


Chemistry 


589 


70 


Life Science/Biology 


586 


61 


Earth/Space Science 


563 


49 


Mathematics 


564 


31 


Engineering/Technology 


507 


7 
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Impacts of the Rural Institutes 

The Outreach Participant Questionnaire asked teachers about their attitudes, beliefs, and 
classroom practices. These items were combined into six composite variables (Appendix A 
contains complete composite definitions and scale reliabilities): 

1 . Attitudes toward standards-based teaching; 

2. Perceptions of pedagogical preparedness; 

3. Perceptions of physics content preparedness; 

4. Use of traditional teaching practices; 

5. Use of investigative teaching practices; and 

6. Use of practices that foster and investigative classroom culture. 

By linking data across years, HRI is able to examine changes in these composite scores for 
participants completing the questionnaire on multiple occasions. Because of the increasing 
number of rural outreach participants that have been in the project for multiple years, HRI was 
able to compare participants’ scores on the composite variables after varying lengths of 
participation. Table 15 shows the number of questionnaires submitted by 1,113 outreach 
participants who have completed at least one questionnaire since the project’s inception. 



Table 15 

Number of Questionnaires Completed by Outreach Participants 





Percent of Outreach Participants 
(N = 1,113) 


1 questionnaire 


59 


2 questionnaires 


30 


3 questionnaires 


12 



These longitudinal data have a nested structure, with time points nested within individual 
participants, who are nested within rural institutes. Statistical techniques that do not account for 
potential grouping effects (e.g., participants in one rural institute all had the same workshop 
experience, while participants in another rural institute all shared a somewhat different workshop 
experience) in nested data structures can lead to incorrect estimates of the relationship between 
independent factors and the outcome. Hierarchical regression modeling^ is an appropriate 
technique for analyzing nested data and was used to examine trends in participants’ composite 
scores. The analysis of these data also controlled for participant demographics (gender, grade 
range taught) and institute size. 

Figure 1 shows the predicted composite scores (i.e., scores adjusted for the control variables) 
relating to the attitudes and perceptions of preparedness across the three data time points from 



^ Bryk, A.S. & Raudenbush, S.W. (1992). Hierarchical linear models: Applications and data analysis methods. 
Newbury Park, CA: Sage Publications. 
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the 901 participants included in these analyses.** These analyses show that participants’ 
perceptions of their pedagogical preparedness were significantly higher after one year of 
AAPT/PTRA professional development, with another significant increase after a second year of 
AAPT/PTRA professional development. These yearly increases correspond to effect sizes** of 
0.22 and 0.20 standard deviations, respectively, with a total effect size of 0.42 standard 
deviations for two years of professional development. 

Participant scores on the perceptions of physics content preparedness composite were 
significantly higher after one year of professional development (an effect size of 0.21 standard 
deviations), but did not increase significantly after a second year of professional development. 
There were no significant changes in participants’ attitudes towards standards-based teaching. 
This outcome is not surprising as participants’ attitudes were high at the outset of their 
involvement in the project. 



Outreach Participant Composite Resuits: 
Attitudes and Perceptions of Preparedness 




Attitudes Toward Pedagogical Physics Preparedness*! 

Standards-Based Teaching Preparedness*!! 



□ Before receiving AAPT/PTRA PD □ After 1 Year of AAPT/PTRA PD ■ After 2 Years of AAPT/PTRA PD 



Composite score after 1 year of PD is significantly higher than composite score before receiving PD (HLM, p < 0.05). 

' Composite score after 2 years of PD is significantly higher than composite score before receiving PD (HLM, p < 0.05). 
^ Composite score after 2 years of PD is significantly higher than composite score after 1 year of PD (HLM, p < 0.05). 

Figure 1 



HRI also examined whether participation in AAPT/PTRA professional development had 
different impacts depending on teachers’ gender and grade-level taught. Females had slightly 
higher scores on the attitudes composite (i.e., more positive attitudes toward standards-based 
teaching) than did males; elementary/middle school teachers tended to have higher scores than 



Some participants submitted incomplete questionnaires, which were dropped from the analyses. 
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high school teachers. These differences did not ehange based upon the amount of AAPT/PTRA 
professional development teaehers reeeived. 

Although there were no differenees in seores on the pereeptions of pedagogieal preparedness 
composite by teaeher gender or grade-level taught, there were differences on the pereeptions of 
eontent preparedness eomposite. On average, male teaehers tended to have greater feelings of 
eontent preparedness than female teaehers. However, this differenee between male and female 
teaehers deereased after partieipation in AAPT/PTRA professional development. (See Figure 2.) 
There was also a differenee on this outeome between elementary/middle sehool teaehers and 
high sehool teaehers; this gap did not ehange over time. 



Outreach Participant Composite Resuits: 
Perceptions of Content Peparedness, by Gender 




Before receiving After 1 year of After 2 years of 

AAPT/PTRA PD AAPT/PTRA PD AAPT/PTRA PD 



□ Females □ Males 



Figure 2 



Results from a teacher eontent knowledge impact study provide additional evidenee that the 
AAPT/PTRA rural project is having a positive impaet on participants’ physics content 
knowledge. A 28-item assessment foeusing on ideas in foree and motion was administered to all 
teachers attending a summer rural institute, once at the beginning of the institute and again at the 
end of the institute. Items for the assessment came from the NSF-supported ATLAST project 
(EHR-0335328) and were seleeted by the AAPT/PTRA rural projeet Pis as aligning with the 
goals of the rural institutes focusing on kinematics and dynamics. The assessment eonsisted of 
three types of items, all of which were set in an instruetional context, measuring teachers’: 

• Knowledge of physios eontent; 

• Ability to use physios eontent knowledge to diagnose student thinking; and 

• Ability to use physios eontent knowledge to make instructional decisions. 

The study found that the AAPT/PTRA rural institutes have had a positive impaet on teaehers’ 
physios eontent knowledge. (See Table 16.) Teaehers soored about 6.5 peroentage points higher 
on the post-test than on the pre-test (an effeot size of 0.60 standard deviations), getting 
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approximately 2 more out of the 28 items eorrect on the post-test than on the pre-test. The study 
also found that these impaets oeeurred regardless of teaeher gender; the same size inerease was 
deteeted for male and female partieipants. 



Table 16 

Teacher Assessment Results 



(N = 192) 


Minimum 


Maximum 


Mean 


Standard 

Deviation 


Pre-test 


17.86 


96.43 


58.48 


20.02 


Post-test* 


21.43 


100.00 


64.99 


20.87 



* Post-test scores significantly higher than pre-test scores (one-tailed dependent 
samples t-test, p < 0.05). 



Interviews with outreaeh partieipants support the findings from the questionnaire and the teaeher 
assessment. Six of the 9 interviewed teaehers indicated that the project had increased their 
physics content knowledge (with four spontaneously mentioning it when asked about impacts in 
general). As two outreach participants said: 

I felt confident going in, but it clarified some things and brought out some 
misconceptions that I had. 

The workshop has increased it [physics content knowledge]. With the kinematics, I got in 
there and was kind of lost on the formulas. It has really helped me get back into that 
mindset of using those things and relating to them. In particular, really dealing with the 
motion, in general, and using Newton ’s laws and applying them to everyday things. I 
found out there were some things I was teaching wrong; it was the right idea, but I was 
giving the wrong example. It cleared up some misunderstandings, I had so I could clear 
it up for the students... the vast quantity of material and activities we went through helped 
me. 



Two teachers also stated that PTRA professional development had an impact on their 
understanding of student learning by teaching them about common student misconceptions. As 
these two teachers stated: 

We talked especially about what misconceptions students might have and how I can 
prevent those in the students. 

They handed out a sheet to us and gave us a text called Five Easy Lessons on physics, 
and those had misconceptions in it that the general public and students had on physics, 
and that was the biggest [impact]. Being able to see those and clarify them before I 
teach the students was good. 

The AAPT/PTRA rural project also appears to be having an impact on how physics is being 
taught. The outreach participant questionnaire asked teachers to describe their teaching in the 
first physics/physical science class of the day during the past school year. (If they did not teach 
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physics/physical science, they were asked to deseribe their teaehing in the first seienee class of 
the day.) Three eomposite variables were ereated from these questions to deseribe teaehing 
praetiees. HRI eompared seores on these eomposites aeross the three time points for the 746 
partieipants who responded for the same type of class in all years in which they submitted data. 
These eomposite seores are shown in Figure 3. 

Partieipants’ report of their frequeney of use of investigative teaehing praetiees inereased 
significantly after one year of AAPT/PTRA professional development (an effeet size of 0.21 
standard deviations). Scores on the investigative eulture composite also inereased signifieantly 
after one year of AAPT/PTRA professional development (an effeet size of 0.19 standard 
deviations). In addition, partieipants’ use of traditional teaehing praetiees was signifieantly 
lower after two years of professional development than prior to partieipation (an effeet size of 
-0.21 standard deviations). In addition to providing evidence of the projeet’s impact, these data 
also support the projeet’s deeision to work with partieipants for three years. 



Outreach Participant Composite Resuits: 
Ciassroom Practices 




Investigative Teaching investigative Cuiture* Traditionai Teaching 
Practices* Practices! 



□ Before receiving AAPT/PTRA PD ■ After 1 Year of AAPT/PTRA PD ■ After 2 Years of AAPT/PTRA PD 



Composite score after 1 year of PD is significantly higher than composite score before receiving PD (HLM, p < 0.05). 
^ Composite score after 2 years of PD is significantly lower than composite score before receiving PD (HLM, p < 0.05). 



Figure 3 



Participants’ teaching practices eomposite seores were also tested for differenees by gender and 
grade range. Elementary/middle sehool teaehers tended to have higher seores on the 
investigative teaehing praetiees eomposite, and lower seores on the traditional teaehing praetiees 
composite, than high sehool teaehers. Female teachers tended to have slightly higher scores on 
the investigative culture eomposite than male teaehers. Although there was no differenee overall 
between males and females on the traditional teaehing praetiees eomposite, female teaehers’ 
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scores decreased significantly over time while male teachers’ scores stayed relatively constant. 
(See Figure 4.) 



Outreach Participant Composite Resuits: 
Traditionai Teaching Practices, by Gender 




Before receiving After 1 year of After 2 years of 

AAPT/PTRA PD AAPT/PTRA PD AAPT/PTRA PD 



□ Females □ Males 



Figure 4 



Interviews with the outreach participants provide further evidence of the project’s impact on 
participants’ teaching. All nine interviewees indicated that the AAPT/PTRA professional 
development had an impact on their physics/physical science teaching, specifically mentioning 
implementing activities and labs from the workshops into their teaching. As two participants 
said: 



It has had a big impact on my teaching. 1 have been able to take the labs, like the ones 
on mechanics and momentum, and it helps me explain it all better to the students. They 
like it too. 

I spend a lot more time with labs and less with lecture. 

The interviewees also described how the changes in their teaching have led to improved student 
learning. All nine participants indicated that their use of AAPT/PTRA materials in their 
classrooms had benefited their students. Eight mentioned that their students enjoyed science 
more by doing all the activities provided by PTRA. Two teachers stated that the students’ 
enjoyment in class increased their confidence in doing physics; another two indicated that their 
students are learning more as a result of the project. In the words of the outreach participants: 

The hands-on activities have made it more enjoyable for them. Their enjoyment is 
transmitted into more willingness to participate and make an attempt to learn. 

My students like science more. I think because I’m more comfortable with the subject 
matter that I can explain it better to [my students]. And the activities give me more to 
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show them as I teach. They can actually see the principle at work and not just talk about 
it. I had fewer labs relating to physics than other aspects I teach, but now in doing 
[PTRA], I have more activities and it gives them a different perspective on science. 

Maybe when they get to high school, they won ’t be so worried about taking physics. 

Projectile motion is difficult for students to understand. PTRA has a model to do that. 

By being able to have students see the half projectile and break it down in the steps in 
this particular problem, helped them understand the projectiles before going through the 
full projectile. And I used their discrepant events hand outs. They have a series of tasks 
that students had to evaluate, ranking tasks. Those are really great because students can 
assess themselves to see “do I really understand this or not” and is a really valuable tool 
for me to use. 

My kids had a better knowledge of how electricity works, the majority are kids that don’t 
understanding anything about how things wire up. This program [CASTLE] has taught 
this group of kids much more than any previous year’s group. They do better on my tests, 
and I think they ’ll do better on the state test. 

In an attempt to more systematieally investigate the impaet of the AAPT/PTRA rural projeet on 
student aehievement, HRI and the AAPT/PTRA rural project leadership developed a study for 
investigating the impact of the project on student achievement in kinematics, dynamics, 
momentum, and energy.'” The study utilized a pre-test/post-test design, with teachers from the 
kinematics and dynamics institute serving as a comparison group for teachers from the 
momentum and energy institute (and vice versa). The 50-item assessment utilized in this study 
was developed by the AAPT/PTRA Rural Project leadership with assistance from HRI. 

Although 90 teachers initially agreed to participate in the study, only 49 returned usable data. 
Some teachers dropped out of the study; others provided only portions of the data needed. 
Although HRI anticipated some drop-outs during the planning of the study, more teachers 
withdrew than was expected. Thus, the statistical power of the study was much lower than 
predicted (i.e., the probability of detecting a significant difference among the groups was smaller 
than hoped for). 

Of the 49 teachers that returned complete data, 27 had participated in a 2004 kinematics and 
dynamics institute, 16 had participated in a 2004 energy and momentum institute and a 2003 
kinematics and dynamics institute, and 6 had participated in only a 2004 momentum and energy 
institute. These 49 teachers administered the assessment to 105 classes, and 1,589 students 
completed both a pre- and post-test. 

Student achievement was examined on two outcomes. The first outcome was performance on 
kinematics and dynamics items; the second outcome was performance on momentum and energy 
items. Teachers were classified into one of three categories: (1) participation in a kinematics 
and dynamics institute only, (2) participation in a momentum and energy institute only, or (3) 



Full results of this study can be found in Banilower, E. R. & Fulp, S. L. (2005). Results of the 2004-05 
AAPT/PTRA Rural Project Student Impact Study. Chapel Hill, NC: Horizon Research, Inc. 
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participation in both institutes. The analyses eontrolled for pre-test seores, student gender, 
student raee/ethnieity (eategorized as white/Asian or non- Asian minority), amount of instruetion 
on the topic, class level (basie, eollege prep, or advaneed), and elass size. 

For the momentum and energy outcome, HRI found no significant differences among students by 
whether or not their teaeher partieipated in an AAPT/PTRA rural institute on momentum and 
energy (regardless of whether the teaeher also partieipated in a kinematies and dynamies 
institute). HRI also found no differences among students on the kinematics and dynamics 
outcome by whether or not their teaeher participated in an AAPT/PTRA rural institute on 
kinematies and dynamics. Table 17 shows the post-test means for students in eaeh group (note; 
means are adjusted for student demographies and pre-test seores and represent the “typieal” 
student aeross all of the teaehers’ classes). 



Table 17 



Average Student Post-Test Scores, 
by Teacher Participation in AAPT/PTRA Rural Institute 





AAPT/PTRA Rural Institutes Attended^ 


2004 Kinematics and 
Dynamics Institute 
Only 


2004 Momentum and 
Energy Institute and 
2003 Kinematics and 
Dynamics Institute 


2004 Momentum and 
Energy Institute 
Only 


Momentum and Energy Score 


48.19 


47.83 


43.72 


Kinematics and Dynamics Score 


52.42 


52.62 


51.06 



No significant differences were found in student post-test scores (controlling for pre-test scores and demographics) 
among the three groups (HLM, p > 0.05). 



Surprisingly, for both outcomes, the amount of instructional time devoted to the topic was not 
related to student scores. These results could be interpreted in a number of ways. One 
interpretation could be that the assessment items are not well aligned with teachers' objectives 
for their courses. Another interpretation could be that the items are aligned, but not sensitive to 
instruction (in other words, the items may not be very good for the purpose of this study). A 
third interpretation could be that only a little instruction is needed to improve student 
performance in these areas, and that there are diminishing returns for further instruction. 
Alternatively, it could be that the teachers’ instruction was just not very effective (the average 
kinematics and dynamics pre-test score was about 43, the post was about 53; the average 
momentum and energy pre-test score was about 41, the post was about 48). 

In regard to the overall study findings, a number of interpretations are plausible. First, because 
of the smaller than expected number of teachers participating, the probability of the study finding 
a significant difference, if one really existed, was rather modest. A second possibility is that 
even though the assessment was aligned with the AAPT/PTRA Institute topics, it was not well 
aligned with teachers’ instructional goals. Another possible explanation is that although the 
institutes are having a positive impact on teachers’ understanding of the content (as evidenced by 
the teacher impact study), the institutes are having less of an impact on teachers’ understanding 
of student learning (the common misconceptions, why people have them, and how to confront 
the misconceptions) and/or their ability to provide high quality instruction. A fourth 
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interpretation eould be that teaehers need more time to praetiee what they learned in the institute 
before positive impaets in student aehievement ean be observed. 



Summary and Recommendations 

The AAPT/PTRA rural projeet has continued to make progress. In its fourth year, the project 
successfully established an additional nine Rural Regional Sites, and provided professional 
development to roughly 700 teachers in the 34 rural summer institutes and follow-up sessions. 
The majority of rural institute participants reached the project’s goal of 36 hours of professional 
development during the year. In addition, the project has greatly increased the rate at which 
participants return for additional years of training (50 percent of participants in sites inaugurated 
in 2003 compared to 15 and 24 percent for sites started in 2001 and 2002, respectively). 

The professional development provided at the rural sites appears to be having a positive impact. 
Outreach participants have higher perceptions of pedagogical preparedness and their reported 
frequency of use of investigative teaching practices has increased. Data from questionnaires, 
interviews, and a teacher content assessment also indicate that the project has had a positive 
impact on participants’ physics content knowledge. 

The key to the project’s success is preparing the PTRAs to provide high-quality professional 
development in outreach workshops. One of the strengths of the project is that most, if not all, 
PTRAs have strong physics backgrounds and a great deal of experience teaching physics prior to 
joining the project. However, leading professional development requires skills above and 
beyond those required to be successful in the classroom. 

Data collected about the AAPT/PTRA Leadership Institute indicate that the project has had an 
impact in this area. PTRAs reported gaining activities and instructional strategies at the 
Leadership Institute that they can use both in their own classrooms and in their outreach 
workshops. PTRAs also reported a positive impact on their preparedness to implement 
professional development (e.g., their preparedness to help outreach teachers develop questioning 
strategies to elicit student understanding). 

Another strength of the project is the collection of instructional resources that has been amassed 
in the creation of the AAPT/PTRA manuals. These manuals are the foundation of the outreach 
workshops, and the outreach participants highly value receiving the activities in them. However, 
even the best activities won’t improve student learning if they are not implemented properly. In 
evaluating a wide array of professional development programs and observing a great deal of 
classroom instruction, HRI has found that teachers often need support to effectively implement 
new instructional materials. It is in this area that HRI offers the following recommendations to 
the project. 

> The project should consider systematically providing additional pedagogical content 
knowledge information with each of the AAPT/PTRA manual activities intended for 
use in the rural institutes. 
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In addition to being familiar with the meehanies of implementing an activity in the 
classroom, teachers should know several things to maximize the effectiveness of an 
instructional activity, including: 

• The specific learning goal(s) addressed by the activity; 

• Where in the instructional sequence the activity should be placed (i.e., what 
instruction students should have prior to and after experiencing the activity); 

• The conceptions/misconceptions students are likely to have prior to experiencing the 
activity, as well as which misconceptions the activity confronts; 

• Likely student responses to the activity (both correct and incorrect); and 

• How to wrap-up the activity to increase the likelihood that student sense-making 
occurs. 

Without this type of information, outreach participants may implement an activity without 
having a clear idea of how the activity adds to the conceptual storyline of a unit, resulting in 
“activity for activity’s sake” (i.e., doing hands-on, but not minds-on). Currently, many 
AAPT/PTRA manuals do not include this type of information with the activities. Although 
some PTRAs may integrate these aspects into their workshops, systematically building this 
information into the AAPT/PTRA manuals would increase the likelihood that all PTRAs do 
so. Given the large number of manuals and activities, the project might consider setting 
aside time at the AAPT/PTRA Leadership Institute for such a task, splitting the most 
commonly used activities up among small groups of PTRAs and having them add the needed 
information. 

In addition, the project should consider providing PTRAs with a template cover sheet for 
activities that would include areas for the above information to be completed during the 
outreach workshop. Providing the PTRAs with such a template would emphasize the need 
for them to address these areas in their outreach workshops, and would provide a mechanism 
for the outreach participants to record this important information. 

> The project should continue to work on developing the PTRAs’ abilities to facilitate 
high-quality professional development. 

Just like students need opportunities to practice and receive feedback on the skills they are 
expected to master, the PTRAs need opportunities to practice and receive feedback on their 
professional development facilitation skills. Providing the PTRAs with these opportunities 
may require decreasing the Institute’s focus on familiarizing PTRAs with classroom 
activities and increasing the focus on practicing and receiving feedback on leadership skills 
(e.g., leading discussions, informally assessing participants’ understanding, helping outreach 
participants examine student work/focus on questioning strategies). These changes may 
require the project leadership to increase their work with the AAPT/PTRA Leadership 
Institute workshop leaders, helping them incorporate what is known from physics education 
research and the research on effective professional development. 



Horizon Research, Inc. 



28 



April 2006 




Appendix A 

Analysis and Reporting of Questionnaire Data 



To facilitate the reporting of large amounts of survey data, and beeause individual questionnaire 
items are potentially unreliable, groups of survey questions that measure similar ideas ean be 
eombined into “eomposites.” Eaeh eomposite represents an important eonstruet related to 
seienee teaehing or professional development. Cronbaeh’s Coeffieient Alpha is a measure of the 
reliability of a eomposite (i.e., the extent to whieh the items appear to be measuring the same 
eonstruet). A Cronbaeh’s Alpha of 0.6 is eonsidered aeeeptable, 0.7 fair, 0.8 good, and 0.9 
exeellent. 

Eaeh eomposite is calculated by summing the responses to the items assoeiated with that 
eomposite and then dividing by the total points possible. In order for the eomposites to be on a 
100-point seale, the lowest response option on eaeh seale was set to 0 and the others were 
adjusted aeeordingly; so for instanee, an item with a seale ranging from 1 to 5 was re-eoded to 
have a seale of 0 to 4. Asa result, someone who marks the lowest point on every item in a 
eomposite reeeives a eomposite score of 0 rather than some positive number. It also assures that 
50 is the true mid-point. The denominator for eaeh eomposite is determined by eomputing the 
maximum possible sum of responses for a series of items and dividing by 100; e.g., a nine-item 
eomposite where eaeh item is on a scale of 0-4 would have a denominator of 0.36. 



PTRA Pre- and Post-Institute Questionnaire Composite Definitions 



Table A-1 

Preparedness to Provide Professional Development 



Questionnaire Composite 


Pre 


Post 


Develop their own understanding of important physics concepts 


Q12ap 


Q8ap 


Understand student thinking and/or common misconceptions related to important physics concepts 


Q12bp 


Q8bp 


Examine science pedagogy/teaching strategies (e.g., white boarding, pair share) and when/why to 
use them 


Q12cp 


Q8cp 


Understand when and why to use a particular activity within their science curriculum 


Q12dp 


Q8dp 


Learn how to examine student work in order to assess student thinking and reflect on classroom 
practice 


Q12ep 


Q8ep 


Identify/develop lessons aligned to learning goals and state and national standards 


Q12fp 


Q8fp 


Develop effective questioning strategies to elicit student understanding 


Q12gp 


Q8gp 


Informally assess student learning 


Q12hp 


Q8hp 


Formally assess student learning 


Q12ip 


Q8ip 








Number of Items in Construct 


9 


9 


Reliability (Cronbach’s Coefficient Alpha) 


0.90 


0.89 
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Outreach Participant Questionnaire Composite Definitions 



Table A-2 

Attitudes Toward Standards-^astd Teaching 



Questionnaire Composite 




Provide concrete experience before abstract concepts. 


Q8ai 


Develop students' conceptual understanding of science. 


Q8bi 


Make connections between science and other disciplines. 


Q8di 


Have students work in cooperative learning groups. 


Q8ei 


Have students participate in appropriate hands-on activities. 


Q8fr 


Engage students in inquiry-oriented activities. 


Q8gi 


Use computers. 


Q8ji 


Engage students in applications of science in a variety of contexts. 


Q8ki 


Use portfolios. 


Q8mi 


Use informal questioning to assess student understanding. 


Q8ni 






Number of Items in Construct 


10 


Reliability (Cronbacb’s Coefficient Alpha) 


0.73 



Table A-3 

Pedagogical Preparedness 



Questionnaire Composite 




Provide concrete experience before abstract concepts. 


Q8ap 


Develop students’ conceptual understanding of science. 


Q8bp 


Take students’ prior understanding into account when planning curriculum and instruction. 


Q8cp 


Make connections between science and other discipline. 


Q8dp 


Have students work in cooperative learning groups. 




Have students participate in appropriate hands-on activities. 


Q8fp 


Engage students in inquiry-oriented activities. 


Q8gp 


Engage students in applications of science in a variety of contexts. 


Q8kp 


Use performance-based assessment. 


Q81p 


Use portfolios. 


Q8mp 


Use informal questioning to assess student understanding. 


Q8np 


Lead a class of students using investigative strategies. 


Q9a 


Manage a class of students engaged in hands-on/project-based work. 


Q9b 


Help students take responsibility for their own learning. 


Q9c 


Recognize and respond to student diversity. 


Q9d 


Encourage students' interest in science. 


Q9e 


Use strategies that specifically encourage participation of females and minorities in science. 


Q9f 


Involve parents in the science education of their students. 


Q9g 






Number of Items in Construct 


18 


Reliability (Cronbacb’s Coefficient Alpha) 


0.91 
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Table A-4 

Physics Content Preparedness (Version A) 



Questionnaire Composite 




Forces and motion 


QlOal 


Energy 


Q10a2 


Light and sound 


Q10a3 


Electricity and magnetism 


Q10a4 


Modern physics (e.g., special relativity) 


Q10a5 


Formulating hypotheses, drawing conclusions, making generalizations 


QlObl 


Experimental design 


Q10b2 


Describing, graphing, and interpreting data 


Q10b3 






Number of Items in Construct 


8 


Reliability (Cronbach’s Coefficient Alpha) 


0.85 



Table A-5 

Physics Content Preparedness (Version B) 



Questionnaire Composite 




Kinematics (i.e., Motion) 




Forces (i.e., gravitational, normal, friction, tension) 




Newton’s Laws 




Linear Momentum 




Energy (i.e., Thermodynamics) 




Energy as a Societal Issue 


KdliHM 


Static Electricity 


KdlilSViM 


Direct Current Circuits 




Formulating hypotheses, drawing conclusions, making generalizations 


MIriliM 


Experimental design 




Describing, graphing, and interpreting data 


Q10b3 






Number of Items in Construct 


II 


Reliability (Cronbach’s Coefficient Alpha) 


0.91 



Table A-6 

Traditional Teaching Practices 



Questionnaire Composite 




Assign science/mathematics homework. 


Q13m 


Answer textbook/worksheet questions 


Q14g 


Review homework/worksheet assignments. 


Q14h 


Take short-answer tests (e.g., multiple choice, true/false, fill-in-the-blank). 


Q14y 






Number of Items in Construct 


4 


Reliability (Cronbach’s Coefficient Alpha) 


0.65 
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Table A-7 

Investigative Teaching Practices 



Questionnaire Composite 




Make formal presentations to the class. 


Q14d 


Engage in hands-on science activities. 


Q14k 


Design or implement their own investigation. 


Q14m 


Work on models or simulations. 


Q14o 


Work on extended science investigations or projects (a week or more in duration). 


Q14p 


Participate in field work. 


Q14q 


Write reflections in a notebook or journal. 


Q14s 


Work on portfolios. 


Q14x 






Number of Items in Construct 


8 


Reliability (Cronbach’s Coefficient Alpha) 


0.76 



Table A-8 

Investigative Classroom Culture 



Questionnaire Composite 




Arrange seating to facilitate student discussion. 


Q13d 


Use open-ended questions. 


Q13e 


Require students to supply evidence to support their claims. 


Q13f 


Encourage students to explain concepts to one another. 


Q13g 


Encourage students to consider alternative explanations. 


Q13h 


Participate in discussions with the teacher to further science understanding. 


Q14b 


Work in cooperative learning groups. 


Q14c 


Share ideas or solve problems with each other in small groups. 


Q14j 






Number of Items in Construct 


8 


Reliability (Cronbach’s Coefficient Alpha) 


0.77 
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Appendix B 



PTRA Pre-Institute Questionnaire 

PTRA Post-Institute Questionnaire 

Rural Institute Participant Questionnaire (Version A) 
Rural Institute Participant Questionnaire (Version B) 




